{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Bayesian methods for performing inference using neural networks\n", "\n", "Tom Charnock\n", "\n", "Institut d'Astrophysique de Paris\n", "


\n", "\n", "
\n", "
\n", " \"Sorbonne\n", "
\n", "
\n", " \"ANR\"\n", "
\n", "
\n", " \"IAP\"\n", "
\n", "
\n", " \"CNRS\"\n", "
\n", "
\n", " \"Aquila\"\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Neural networks for parameter regression" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## What is a neural network?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "\n", "
\n", "
An arbitrary, non-linear function $(\\mathscr{f}:\\mathbb{R}^{\\bf d}\\rightarrow\\mathbb{R}^\\boldsymbol{t})$ with fittable parameters $(\\boldsymbol{w})$
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Let's talk about parameter estimation by regression" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "\n", "
\n", "
Take in data and predict model parameters
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Extremely popular for cosmological parameter estimation\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Only provide intrinsically biased estimates!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Some stats" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### A model\n", "The generator $({\\bf d}\\in\\mathcal{M}(\\boldsymbol{\\theta}))$ of data $({\\bf d})$ with model parameters $(\\boldsymbol{\\theta})$ " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### The likelihood and the posterior" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
\n", " Likelihood $\\mathcal{L}({\\bf d}|\\boldsymbol{\\theta}^*)$
What is the probability that the model generates data ${\\bf d}$ given a set of model parameters $\\boldsymbol{\\theta}^*$?

\n", " Posterior $\\mathcal{P}(\\boldsymbol{\\theta}|{\\bf d}^*)$
What is the probability that model parameters $\\boldsymbol{\\theta}$ generate the data ${\\bf d}^*$?\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Fisher information\n", "
\n", "
\n", " How much information does the data ${\\bf d}$ contain about the model parameters $\\boldsymbol{\\theta}$?\n", "
\n", "
\n", " $${\\bf F}_{\\alpha\\beta} = \\left.\\left\\langle\\frac{\\partial^2\\ln\\mathcal{L}({\\bf d}|\\boldsymbol{\\theta})}{\\partial\\theta_\\alpha\\partial\\theta_\\beta}\\right\\rangle\\right|_{\\boldsymbol{\\theta}=\\boldsymbol{\\theta}^*}$$\n", "
\n", " For a Gaussian\n", "
\n", " $${\\bf F}_{\\alpha\\beta} = \\frac{\\partial\\mu}{\\partial\\theta_\\alpha}^T{\\bf C}^{-1}\\frac{\\partial\\mu}{\\partial\\theta_\\beta}.$$\n", "
\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Maximum likelihood estimates" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
\n", " Which value of the model parameters $\\boldsymbol{\\theta}$ is most likely to generate the observed population of data $\\{{\\bf d}\\}$?\n", "
\n", " \n", " $$\\mathcal{L}(\\{{\\bf d}\\}|\\boldsymbol{\\theta}_\\textsf{MLE})=\\underset{\\boldsymbol{\\theta}\\in\\boldsymbol{\\Theta}}{\\textsf{Sup}}~\\mathcal{L}(\\{{\\bf d}\\}|\\boldsymbol{\\theta})$$\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Intrinsically biased estimators" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "
\n", " Intrinsically biased estimators

A summary of the parameters which is conditioned on the wrong likelihood.

\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Training a network" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
\n", " Neural network is a model $\\mathcal{M}(\\boldsymbol{w}, {\\bf d})$ with parameters $\\boldsymbol{w}$ and initial conditions ${\\bf d}$.\n", "

\n", " Training amounts to finding the maximum likelihood estimates of the weights for a chosen cost function $\\boldsymbol{l}$.

\n", "
\n", "


\n", "
\n", " $$\\mathcal{L}(\\boldsymbol{l}\\,|\\boldsymbol{w}_\\textsf{MLE}, {\\bf d})=\\underset{\\boldsymbol{w}\\in\\boldsymbol{W}}{\\textsf{Sup}}~\\mathcal{L}(\\boldsymbol{l}\\,|\\boldsymbol{w}, {\\bf d})$$\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Why do neural networks give intrinsically biased estimates?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### The likelihood surface is not knowably convex (or concave)\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", "
The likelihood surface is conditional on the architecture and the data
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### There is no known *correct* minimum which is complete for all data\n", " \n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Even *well converged* networks are unknowably biased" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## In general outputs are highly informative\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " Outputs can look like model parameter estimates\n", "
\n", "
\n", " They cannot be trusted as true predictions of the parameters\n", "
\n", "
\n", " The can be used as informative summaries!\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## How can we use a neural network safely?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Build it into the physical model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Likelihood-free inference using machine learning" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Approximate Bayesian computation\n", "
\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Why even introduce a neural network?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## The curse of dimensionality\n", "
\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Inadequate sampling\n", "\n", "Impossibly large numbers of simulations become necessary to correctly sample the posterior\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# We need to do some compression" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Any pretrained neural network *could* be used" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### Is there a well motivated choice?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Information maximising neural networks\n", "
\n", "
\n", " Which function $\\mathscr{f}: \\mathbb{R}^{\\bf d}\\to \\mathbb{R}^\\boldsymbol{\\theta}$ maximises the Fisher information of the summaries ${\\bf x}$ from that function?\n", "
\n", "
\n", "
\n", " $$\\mathcal{L}({\\bf d}|\\boldsymbol{\\theta})\\to\\mathcal{L}({\\bf x}|\\boldsymbol{\\theta}, {\\bf d})$$\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Fisher information of the network summaries\n", "\n", "#### Simulations $\\{{\\bf d}_i|i\\in[1, n_{\\bf d}]\\}$ at a single parameter value $\\boldsymbol{\\theta}^*$\n", "\n", "#### Seed matched simulations $\\{{\\bf d}_i^{\\pm}|i\\in[1, n_{\\bf p}]\\}$ at perturbed fiducial paremeters $\\Delta\\boldsymbol{\\theta}^\\pm=\\boldsymbol{\\theta}\\pm\\boldsymbol{\\delta}$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Calculate the covariance\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Calculate the derivative of the mean of the summaries with respect to the parameters\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Calculate the Fisher information\n", "\n", "$${\\bf F}_{\\alpha\\beta}=\\frac{\\partial\\mu_\\mathscr{f}}{\\partial\\theta_\\alpha}^T{\\bf C}^{-1}_\\mathscr{f}\\frac{\\partial\\mu_\\mathscr{f}}{\\partial\\theta_\\beta}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### This Gaussian form forces the summaries to be Gaussianised\n", "
\n", "
Makes a non-linear mapping of (non-Gaussian) data to the compressed set of Gaussian summaries
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Optimise the tunable parameters of the network such that $\\textsf{ln}|{\\bf F}_{\\alpha\\beta}|$ is maximised!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Once converged the network compresses *and* Gaussianises the data *without* losing information*\n", "



\n", "*in the optimal case..." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### And we get free maximum likelihood estimates...\n", "

\n", "$$\\boldsymbol{\\theta}^\\textsf{MLE}_\\alpha=\\boldsymbol{\\theta}_\\alpha^*+{\\bf F}_{\\alpha\\beta}^{-1}{\\bf C}_\\mathscr{f}^{-1}\\frac{\\partial\\mu_\\mathscr{f}}{\\partial\\theta_\\beta}({\\bf x}-\\mu_\\mathscr{f})$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### And we can get free approximations of the posterior\n", "

\n", "$$\\textsf{Cov}[\\boldsymbol{\\theta}_\\alpha^\\textsf{MLE},\\boldsymbol{\\theta}_\\beta^\\textsf{MLE}] \\ge {\\bf F}_{\\alpha\\beta}^{-1}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Or do likelihood-free inference" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### We pass our observed data through the network
(put it to the side)\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Draw simulations (cleverly?) from the prior model parameters, $\\mathcal{P}(\\boldsymbol{\\theta})$ and pass them through the network\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Measure the difference between the summarised simulations and the observed summary\n", "
\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Compare distance between observed summaries and simulation summaries and select results within $\\epsilon$

\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Conclusions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Neural networks are not to be trusted\n", "\n", "### They can make trusty companions - when the correct framework is introduced\n", "\n", "### Using statistics we can build them into the forward model to give us lossless summaries" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Great, but is there something better?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## We could infer the neural network..." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "## without *any* training data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Our observation\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## What does our observation actually look like\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### What does our observation actually actually look like\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What do we need to infer the parameters of a neural network?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Neural physical engines\n", "\n", "Build networks using physical principles.\n", "\n", "- Reduces number of parameters\n", "\n", "- Increases computational efficiency\n", "\n", "- Decreases overfitting\n", "\n", "- Improves interpretability" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Convolutions for translational invariance\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Architecture suited to the data\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Use only causally relevant features\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Use only informative features\n", "
\n", "\n", "
\n", "github:multipole_kernels" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Neural density estimators" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Halo mass distribution function is a smooth function of mass given a density environment" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Use a mixture density network" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Our neural density estimator\n", "$$\\begin{align*}\n", " {\\tiny n(M|\\delta) =}&{\\tiny \\sum_i^N\\alpha(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\alpha}})\\mathcal{N}\\left(\\mu(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\mu}}), \\sigma(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\sigma}})|M\\right),}\\\\\n", " {\\tiny =}&{\\tiny \\sum_i^N\\frac{\\alpha(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\alpha}})}{\\sqrt{2\\pi\\left(\\sigma(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\sigma}})\\right)^2}}\\exp\\left[-\\frac{\\left(\\log(M) - \\mu(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\mu}})\\right)^2}{2\\left(\\sigma(\\boldsymbol{\\psi}, \\boldsymbol{\\theta}_i^{\\boldsymbol{\\sigma}})\\right)^2}\\right],}\\end{align*}\n", "$$\n", "
\n", " \n", "
\n", "
\n", " $$\n", " \\begin{align*}\n", "{\\tiny\\alpha_i} & {\\tiny=\\text{softplus}(w_{i}^\\alpha\\boldsymbol{\\psi}+b^\\alpha_i),}\\\\\n", "{\\tiny\\mu_i }&{\\tiny = \\left\\{\\begin{array}{ll}\n", " w_{i}^\\mu\\boldsymbol{\\psi}+b^\\mu_i&i=0\\\\\n", " \\textrm{Max}\\left[0,~ w_{i}^\\mu\\boldsymbol{\\psi}+b^\\mu_i\\right]+\\mu_{i-1}&i>0\\\\\n", " \\end{array}\\right.,}\\\\\n", "{\\tiny\\sigma_i} &{\\tiny = \\text{softplus}(w_{i}^\\sigma\\boldsymbol{\\psi}+b^\\sigma_i)\\;,}\n", " \\end{align*}\n", " $$\n", "
\n", "
\n", "2 Gaussians with
${\\Tiny\\boldsymbol{b}^\\alpha \\to \\boldsymbol{b}^\\alpha +\\log(10^{-3})}$, ${\\Tiny b_0^\\mu \\to b_0^\\mu + \\log\\left(2\\times10^{12}\\right)}$ and ${\\Tiny\\boldsymbol{b}^\\sigma\\to \\boldsymbol{b}^\\alpha + \\log(10^3)}$.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Likelihood" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Poisson likelihood of the observed catalogue given the forward model\n", "

\n", "$$\\begin{align*}\n", " {\\Tiny\\mathcal{L} =}&{\\Tiny \\sum_{j\\in\\textsf{catalogue}}\\log\\left[\\sum_i^N\\frac{\\alpha_{i,j}}{\\sqrt{2\\pi\\sigma_{i,j}^2}}\\textsf{exp}\\left[-\\frac{\\left(\\textsf{ln}(M_j) - \\mu_{i,j}\\right)^2}{2\\sigma_{i,j}^2}\\right]\\right]}\\\\\n", " &{\\Tiny - V\\sum_{j\\in\\textsf{voxels},i=1}^N\\frac{\\alpha_{i,j}}{2}\\textsf{exp}\\left[\\frac{\\sigma_{i,j}^2}{2}\\right]\\textsf{erfc}\\left[\\frac{\\textsf{ln}\\left(M_\\textsf{th}\\right) - \\mu_{i,j} - \\sigma_{i,j}^2}{\\sqrt{2\\sigma_{i,j}^2}}\\right].}\n", " \\end{align*}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# HMCLET" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### We want to infer the weights of the neural bias model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Hamiltonian Monte Carlo\n", "
\n", "Introduce momentum ${\\small {\\bf p}\\leftarrow\\mathcal{N}({\\bf 0},{\\bf M})}$ and solve Hamilton's equations\n", "

\n", "$$\\begin{align*}\n", "{\\Tiny\\mathcal{H}(\\boldsymbol{\\theta}, {\\bf p}) }&{\\Tiny~= \\mathcal{V}(\\boldsymbol{\\theta}) + \\mathcal{K}({\\bf p})}\\\\\n", "&{\\Tiny~=\\mathcal{L}(\\boldsymbol{\\theta}|\\boldsymbol{\\delta})-\\textsf{ln}\\left[\\pi(\\boldsymbol{\\theta})\\right]+\\frac{1}{2}{\\bf p}^T{\\bf M}^{-1}{\\bf p}}\n", "\\end{align*}$$\n", "Accept samples according to probability\n", "\n", "$$\\begin{equation*}\n", " {\\Tiny\\alpha = \\textsf{Min}\\left[\\textsf{exp}\\left(\\Delta\\mathcal{H}\\right), 1\\right]}\n", "\\end{equation*}$$\n", "
\n", "
\n", "

\n", " \n", "
\n", "
\n", "
\n", "Evolve using ${\\Tiny\\dot{\\boldsymbol{\\theta}}= {\\bf M}^{-1}{\\bf p}}$ and ${\\Tiny\\dot{\\bf p}= -\\nabla\\mathcal{V}(\\boldsymbol{\\theta})}$\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Acceptance is guaranteed if ODEs are perfectly solved" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "# Use leapfrog algorithm" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Leapfrog algorithm\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Acceptance criterion\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## How does this work for neural networks?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Likelihood surface is extremely flat and highly degenerate\n", "
\n", "\n", "\n", "Nearly impossible to know the mass matrix, ${\\bf M}$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Use second order geometric information\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Quasi-Newtonian inverse Hessian\n", "\n", "$$\\begin{equation*}\n", " {\\Tiny{\\bf B}^*=\\left(\\mathbb{I}-\\frac{\\boldsymbol{s}_i\\boldsymbol{y}_i^T}{\\boldsymbol{y}_i^T\\boldsymbol{s}_i}\\right){\\bf B}_i\\left(\\mathbb{I}-\\frac{\\boldsymbol{y}_i\\boldsymbol{s}_i^T}{\\boldsymbol{y}_i^T\\boldsymbol{s}_i}\\right)+\\frac{\\boldsymbol{s}_i\\boldsymbol{s}_i^T}{\\boldsymbol{s}_i^T\\boldsymbol{y}_i},}\n", " \\end{equation*}\n", "$$\n", "with ${\\Tiny\\boldsymbol{s}_i=\\boldsymbol{\\theta}^*-\\boldsymbol{\\theta}_i}$ and ${\\Tiny\\boldsymbol{y}_i=\\nabla\\mathcal{L}(\\boldsymbol{\\theta}^*|\\boldsymbol{\\delta})-\\nabla\\mathcal{L}(\\boldsymbol{\\theta}_i|\\boldsymbol{\\delta})}$.\n", "\n", "#### Both $\\boldsymbol{s}$ and $\\boldsymbol{y}$ come for free with the leapfrog algorithm" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## We can now successfully traverse the likelihood surface\n", "\n", "The momenta and trajectories get rescaled by the Hessian providing a surface which is not as highly degenerate or flat" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## How well does it work?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Sample the weights of the neural bias model\n", "
\n", "\n", "\n", "✓ Weight values are properly sampled after burn in
\n", "✓ NPE acts as a contrast enhancer" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### See the effects of the neural physical engine\n", "
\n", "\n", "\n", "✓ Non-local information is used to improve fit" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# And what does the halo mass distribution function look like?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Halo mass distribution function from neural bias model\n", "
\n", "\n", "✓ Fits data!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Second conclusion" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### We can infer neural networks built for physics\n", "\n", "### Highly efficient and bias free" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Take home message" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Stop doing machine learning, think, then start doing machine learning again!" ] } ], "metadata": { "celltoolbar": "Diaporama", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 4 }